The expression and experience of human behavior are complex, multimodal and characterized by individual and contextual heterogeneity and variability. Speech and spoken language communication cues offer an important means for measuring and modeling human behavior. Observational research and practice across a variety of domains from commerce to healthcare rely on speech and language-based informatics for crucial assessment and diagnostic information and for planning and tracking response to an intervention.
We’ve pioneered a field, Behavioral Signal Processing, based on over a decade’s worth of award-winning and patented research, to automatically detect information that is encoded in the human voice from audio and measure the quality of human interaction. It is an emerging discipline that bridges engineering with behavioral sciences and aims to quantify and interpret human interaction and communication through the use of engineering and computing innovations.
Behavioral Signal Processing focuses on gathering, analyzing and modeling multimodal behavior signals, both overtly and covertly expressed. This builds upon foundational efforts in speech/language/audio/visual/physiological signal processing. Critically, in addition to processing objectively-specified behavioral content in richer ways (e.g., what someone said and did), Behavioral Signal Processing entails automating a host of subjectively-specified entities such as those related to socio-emotional states of people (e.g., how negative or frustrated a person is; politeness; engagement etc).
Human behavior expression and experience are inherently multimodal, and characterized by vast individual and contextual heterogeneity.The confluence of sensing, communication, and computing is allowing access to data in diverse forms and modalities that is enabling us to understand and model human behavior in ways that were unimaginable even a few years ago. No domain exemplifies these opportunities more than that related to human health and well being. Behavioral Signal Processing advances can enable not only new possibilities for gathering data in a variety of settings–from laboratory and clinics to free-living conditions―but in offering computational models to advance evidence-driven theory and practice.
Further Reading
Prof. Shrikanth (Shri) Narayanan
Signal Analysis and Interpretation Laboratory (SAIL)
University of Southern California
Whitepaper
Measuring the how and why in conversational data with artificial intelligence and deep learning